perf(parquet): use Cursor in ZSTDCodec to avoid Vec alloc and copy by MassivePizza · Pull Request #10345 · apache/arrow-rs

MassivePizza · 2026-07-15T09:58:43Z

Which issue does this PR close?

N/A

Rationale for this change

Remove needless allocs and copies.

What changes are included in this PR?

Inline zstd::bulk:: methods that returned a temporary Vec.
Reserve space and write to output_buf directly using a Cursor instead.

Are these changes tested?

Should be covered by existing tests.

Are there any user-facing changes?

None, although there is an implementation detail to consider; Cursor cannot be limited when writing, so this change technically allows compress/decompress to write more data than before, based on excess Vec capacity.
Vec::with_capacity is not required to allocate exactly, so this should be just fine.

Jefffrey · 2026-07-15T11:58:27Z

run benchmark compression
env:
BENCH_FILTER: ZSTD

adriangbot · 2026-07-15T12:04:38Z

🤖 Arrow criterion benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                           compress-vec-begone                    main
-----                                           -------------------                    ----
compress ZSTD(ZstdLevel(1)) - alphanumeric      1.00    920.1±0.50µs        ? ?/sec    1.02    938.5±0.54µs        ? ?/sec
compress ZSTD(ZstdLevel(1)) - words             1.00      2.7±0.02ms        ? ?/sec    1.00      2.7±0.02ms        ? ?/sec
decompress ZSTD(ZstdLevel(1)) - alphanumeric    1.00   1003.9±1.58µs        ? ?/sec    1.02   1021.8±0.59µs        ? ?/sec
decompress ZSTD(ZstdLevel(1)) - words           1.00    978.9±1.57µs        ? ?/sec    1.02   1001.0±1.68µs        ? ?/sec

Resource Usage

base (merge-base)

Metric	Value
Wall time	45.0s
Peak memory	10.4 MiB
Avg memory	9.0 MiB
CPU user	42.7s
CPU sys	0.0s
Peak spill	0 B

branch

Metric	Value
Wall time	50.0s
Peak memory	12.5 MiB
Avg memory	9.2 MiB
CPU user	46.0s
CPU sys	0.0s
Peak spill	0 B

File an issue against this benchmark runner

Jefffrey · 2026-07-15T17:13:41Z

run benchmark compression
env:
BENCH_FILTER: ZSTD

adriangbot · 2026-07-15T17:19:44Z

🤖 Arrow criterion benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                           compress-vec-begone                    main
-----                                           -------------------                    ----
compress ZSTD(ZstdLevel(1)) - alphanumeric      1.00    917.7±0.56µs        ? ?/sec    1.03    942.5±2.87µs        ? ?/sec
compress ZSTD(ZstdLevel(1)) - words             1.00      2.7±0.01ms        ? ?/sec    1.00      2.7±0.00ms        ? ?/sec
decompress ZSTD(ZstdLevel(1)) - alphanumeric    1.00   1003.6±0.59µs        ? ?/sec    1.02   1024.5±2.90µs        ? ?/sec
decompress ZSTD(ZstdLevel(1)) - words           1.00    978.2±1.41µs        ? ?/sec    1.02    998.9±1.90µs        ? ?/sec

Resource Usage

base (merge-base)

Metric	Value
Wall time	45.0s
Peak memory	10.4 MiB
Avg memory	9.0 MiB
CPU user	42.8s
CPU sys	0.0s
Peak spill	0 B

branch

Metric	Value
Wall time	50.0s
Peak memory	12.7 MiB
Avg memory	9.2 MiB
CPU user	46.0s
CPU sys	0.0s
Peak spill	0 B

File an issue against this benchmark runner

Jefffrey

None, although there is an implementation detail to consider; Cursor cannot be limited when writing, so this change technically allows compress/decompress to write more data than before, based on excess Vec capacity.
Vec::with_capacity is not required to allocate exactly, so this should be just fine.

could you elaborate on this a bit?

MassivePizza · 2026-07-20T13:08:35Z

And to explain the Cursor "issue" a bit better; it exposes the entire spare capacity of Vec to the decompressor, which can lead to decompression of more than we request.
decompress(...) -> Vec can exhibit this as well if the internal Vec::with_capacity(x) allocates more than x.
It's just more likely that we have excess spare capacity after output_buf.reserve().

Jefffrey · 2026-07-23T01:49:46Z

run benchmark compression
env:
BENCH_FILTER: ZSTD

adriangbot · 2026-07-23T01:55:04Z

🤖 Arrow criterion benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                           compress-vec-begone                    main
-----                                           -------------------                    ----
compress ZSTD(ZstdLevel(1)) - alphanumeric      1.00    922.4±0.85µs        ? ?/sec    1.01    934.0±1.46µs        ? ?/sec
compress ZSTD(ZstdLevel(1)) - words             1.00      2.7±0.01ms        ? ?/sec    1.00      2.7±0.01ms        ? ?/sec
decompress ZSTD(ZstdLevel(1)) - alphanumeric    1.00   1003.9±0.73µs        ? ?/sec    1.02   1020.4±0.74µs        ? ?/sec
decompress ZSTD(ZstdLevel(1)) - words           1.00    983.7±2.67µs        ? ?/sec    1.01    998.1±1.25µs        ? ?/sec

Resource Usage

base (merge-base)

Metric	Value
Wall time	50.0s
Peak memory	10.8 MiB
Avg memory	8.5 MiB
CPU user	42.9s
CPU sys	0.0s
Peak spill	0 B

branch

Metric	Value
Wall time	50.0s
Peak memory	12.8 MiB
Avg memory	9.4 MiB
CPU user	47.0s
CPU sys	0.0s
Peak spill	0 B

File an issue against this benchmark runner

Jefffrey · 2026-07-23T08:14:03Z

thanks @MassivePizza

mapleFU · 2026-07-28T05:31:26Z

Maybe not a problem, as described in #10448 :

It's "Free" For data page v1, since it will finally shrink_to_fit
For Data page v2, shrink_to_fit would not called, and this will introducing some memory overhead

github-actions Bot added the parquet Changes to the parquet crate label Jul 15, 2026

perf: use Cursor in ZSTDCodec to avoid Vec alloc and copy

02523a9

MassivePizza force-pushed the compress-vec-begone branch from 47a9e23 to 02523a9 Compare July 15, 2026 10:00

nit: rename vars

0ecd05b

This comment was marked as duplicate.

Sign in to view

Jefffrey reviewed Jul 15, 2026

View reviewed changes

Comment thread parquet/src/compression.rs Outdated

This comment was marked as duplicate.

Sign in to view

MassivePizza changed the title ~~perf: use Cursor in ZSTDCodec to avoid Vec alloc and copy~~ perf(parquet): use Cursor in ZSTDCodec to avoid Vec alloc and copy Jul 17, 2026

MassivePizza added 2 commits July 17, 2026 12:41

use zstd::zstd_safe re-export

d0f8513

Merge branch 'main' into compress-vec-begone

98fee5a

Jefffrey reviewed Jul 18, 2026

View reviewed changes

Comment thread parquet/src/compression.rs

Comment thread parquet/src/compression.rs Outdated

clarify upper_bound comment

02bd80c

MassivePizza added 2 commits July 20, 2026 15:35

restore get_frame_content_size and remove upper_bound

d0d52dd

Merge branch 'main' into compress-vec-begone

deabfbd

This comment was marked as duplicate.

Sign in to view

Jefffrey added the performance label Jul 23, 2026

Jefffrey approved these changes Jul 23, 2026

View reviewed changes

Jefffrey merged commit 26aff59 into apache:main Jul 23, 2026
16 checks passed

MassivePizza mentioned this pull request Jul 27, 2026

[X-3344] Backport optimization to 58.3 branch massive-com/arrow-rs#2

Open

mapleFU mentioned this pull request Jul 28, 2026

Parquet: Should DataPageV2 shrink_to_fit? #10448

Open

MassivePizza deleted the compress-vec-begone branch July 28, 2026 12:03

MassivePizza mentioned this pull request Jul 29, 2026

Avoid over-reserving Vecs in compression codecs #10483

Open

Uh oh!

Conversation

MassivePizza commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Jefffrey commented Jul 15, 2026

Uh oh!

This comment was marked as duplicate.

adriangbot commented Jul 15, 2026

Uh oh!

Uh oh!

Jefffrey commented Jul 15, 2026

Uh oh!

This comment was marked as duplicate.

adriangbot commented Jul 15, 2026

Uh oh!

Jefffrey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

MassivePizza commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Jefffrey commented Jul 23, 2026

Uh oh!

This comment was marked as duplicate.

adriangbot commented Jul 23, 2026

Uh oh!

Uh oh!

Jefffrey commented Jul 23, 2026

Uh oh!

mapleFU commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

MassivePizza commented Jul 15, 2026 •

edited

Loading

MassivePizza commented Jul 20, 2026 •

edited

Loading